fix(cassandra): a Query of a partition with no items left answers no items - #611

On Cassandra, a Query of a partition whose items have all been deleted now answers no items. Before, it answered 500 InternalServerError, and a GSI added by UpdateTable to a table holding such a partition stayed CREATING. A Query that reads other rows without an item, such as a TTL claim on a key being written, now answers the items around them instead of 500.

This PR fixes no open issue.

What happened

DeleteItem sets the partition's static column partition_max_delete_timestamp. When the last item of a partition is deleted, Cassandra still answers a read of that partition with one row that carries only the static columns and no item_data. Query parsed that row as an item and failed with Column or Udt property 'item_data' is empty (server log), answering 500. The GSI backfill, which scans the base table, read the same row, failed the same way, and retried for as long as the index existed.

A TTL claim or a transaction prepare on an absent key writes a data row with only prepared_txn_id set, also without item_data. Query answered 500 on those rows too.

A Query with a sort key condition or an ExclusiveStartKey, an index Query and a hash-only table do not see the static-only row: Cassandra returns it only when the read restricts no clustering column, an index Query reads the index table, and a hash-only table keeps the delete timestamp as a regular column of the item's own row.

What the service answers

Measured on 2026-10-26 at 06:48, 06:52, 06:59, 07:09, 08:00 and 14:45 UTC in us-east-1, each case twice with the same answer.

• A Query of a partition whose items have all been deleted answers Count 0 with no items, in six shapes (plain, ConsistentRead, Limit 1, ScanIndexForward false, Select COUNT, a sort key condition) and on a global secondary index. GetItem of a deleted item answers no Item.
• The same holds with a FilterExpression, a ProjectionExpression, Select SPECIFIC_ATTRIBUTES, an ExclusiveStartKey before, inside or after the deleted items, and when the items were deleted by TransactWriteItems or BatchWriteItem. Such a Query consumes 0.5 capacity units, or 1.0 with ConsistentRead.
• A partition with one item left answers that item. A hash-only table whose item was deleted answers nothing to Query, GetItem and Scan.
• A GSI added by UpdateTable to a table with an emptied partition and one live item becomes ACTIVE (after 520 s and 1,011 s) and answers the live item only.

The change

• crates/storage-cassandra/src/data/query.rs: a row without item_data is skipped. Query asks Cassandra for LIMIT limit + 1 rows, so skipped rows take places in the read. Query now reports more pages when the read filled its LIMIT, counting the rows Cassandra returned rather than the items kept, so a page that skipped rows can be short but carries a LastEvaluatedKey, and the engine reads on from its last item. A read in which no row holds an item has no item to continue from; Query reads that one again with twice the LIMIT, and doubles it again while every row read holds no item, so a run of R such rows costs about log2(R / Limit) extra reads.
• crates/storage-cassandra/src/data/ddl.rs: the GSI backfill skips such a row. It already counts raw rows for its page size and takes its cursor from the row's pk, which a static-only row carries.

Tests

• tests/test_query_emptied_partition.py: 17 dual-target wire tests built from the measurements. The index test waits up to 1,500 s for the index against the service, which took up to 1,011 s, and 120 s against extenddb.
• crates/storage-cassandra/tests/direct/query_rows_without_item.rs: 4 tests. One reads an emptied partition. The others write rows with only prepared_txn_id set (with CQL): before, between and after ten items, paged both ways with page sizes 1 to 5 and 9 to 11 and with no Limit; one such row in front of 3,000 items; and a run of 3,000 such rows in front of two items.

Run locally against Cassandra 4.1 (one node, PasswordAuthenticator) on 2026-10-26:

• the direct tests: 4 passed. With the first commit's query.rs (skip only), 3 fail (Limit 1 returns no item and no LastEvaluatedKey); with main's, all 4 fail (500).
• cargo test -p extenddb-storage-cassandra --test direct_integration: 153 passed. --test ttl_integration: 26 passed, 1 ignored, and test_ttl_enable_rejects_asynchronous_gsi fails here with main's storage code as well.
• the wire file, with the runtime settings devtools/run-tests applies in CI: 17 passed at the head. With main's binary, 9 fail (the eight Query tests on a partition with no item left get 500, and the index added later stays CREATING) and 8 pass.
• the 32 Query cases measured at 14:45 UTC, replayed against a local table: the head answers as the service in all 32 on Cassandra and on SQLite. main's binary on Cassandra answers 24 of them with 500.
• the wire tests whose names match query, scan, delete, gsi, index, update_table, backfill, ttl, transact, batch, backup, export, restore or lsi (652 tests): 38 fail at the head, and each of them fails on main too, from the local single-node setup and the multi-attribute GSI gap below.
• a TTL table, one partition of 300 items, two writers putting new keys into it for 60 s and one reader paging Query through it: 1,021 full reads at the head, each with every item. With the first commit's query.rs, 9 of 1,162 reads ended at 128 items with no LastEvaluatedKey; on main, 326 of 983 answered 500.
• one row with only prepared_txn_id set in front of 3,000 items: a Query with Limit 1 and with Limit 100 costs one Cassandra read (nodetool tablestats), as for the same partition without the row; with no Limit, 4 reads in both partitions.
• the wire file on SQLite: 17 passed. CI also runs it on PostgreSQL and MongoDB.
• cargo +1.97.0 fmt --all -- --check and cargo +1.97.0 clippy --all-targets -- -D warnings: clean. cargo test --workspace --exclude extenddb-storage-cassandra: 1,794 passed on stable and on 1.88.0. cargo test -p extenddb-storage-cassandra --lib: 61 passed.

Not changed here

These answer the same on main:

• Scan drops rows without item_data after Cassandra applies its LIMIT, and sets "more pages" from the items kept, so a Scan ends early with no LastEvaluatedKey after an emptied partition. 300 live single-item partitions and 30 emptied ones: a Scan with no Limit returns 119 items. CreateBackup, the TTL backfill and audit, ItemCount and TableSizeBytes, and export read through the same scan: 1,100 live items and 20 emptied partitions restore as 987 items.
• The GSI backfill pages with token(pk) > token(last pk), so a partition larger than its 500-row page loses its tail: a GSI added to a table with one 600-item partition holds 500 entries.
• A Query on a GSI with more than one sort key attribute answers 500 (Order by currently only supports the ordering of columns following their declared order in the PRIMARY KEY). fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429 changes this area.
• A Query with a sort key condition and an ExclusiveStartKey answers 500 (Cassandra: a Query with a sort key condition and an ExclusiveStartKey answers InternalServerError #488), so a BETWEEN Query fails on its second page: the read has two lower bounds on the sort key column, which Cassandra refuses (More than one restriction was found for the start bound).
• When a page reaches its Limit on the partition's last item, extenddb returns a LastEvaluatedKey and the next page is empty. The engine's page loop in read_helpers.rs, which every backend uses, sets it; measured on Cassandra and SQLite. At 14:45 UTC the service answered such a page with no LastEvaluatedKey in 46 of 48 cases (partitions of one, two and three items, with and without deleted items, both read consistencies, both directions), and with a key and an empty next page in the other 2 (second round, descending, two deleted items before the live one).

Merging

git merge-tree --write-tree against the 16 open PRs that change query.rs, ddl.rs or tests/direct_integration.rs: #527 and #395 add their own lines to the mod list of tests/direct_integration.rs, so each gains a conflict there (keep both sides); #395 conflicts with main already in other files, as do #588 and #474. The other 12 merge clean.